My review for this work remains the same following author response because I believe that the authors have demonstrated this work to be of high quality and relevance to the NeurIPS community. Originality: Although the algorithms used to synthesize the metamers themselves are nothing new, the work is a novel combination of previous approaches and techniques, and the analysis approach gives these methods a fresh perspective that leads to good insights. Quality: The work is of high quality and is a complete piece of work that will advance our understanding of the relationships between architecture, task and training in determining representational similarity between networks and humans (as well as between networks). Clarity: The paper is overall clear, though some details could use a bit of clarifying (what was the threshold for satisfactory termination of synthesis? Significance: This work builds on theoretical and experimental work from neuroscience used to analyze how well models of perceptual systems capture the representation within the human brain by synthesizing stimuli that match the responses of some part of the model completely and using human subjects to validate that the original and matched stimulus are in fact the same.

Dear authors, congrats on the acceptance-- this paper was discussed extensively, the the reviewers provided multiple comments and feedback-- please do take the feedback in the reviews into account when preparing your final manuscript.

Deep neural networks have been embraced as models of sensory systems, instantiating representational transformations that appear to resemble those in the visual and auditory systems. To more thoroughly investigate their similarity to biological systems, we synthesized model metamers – stimuli that produce the same responses at some stage of a network's representation. We generated model metamers for natural stimuli by performing gradient descent on a noise signal, matching the responses of individual layers of image and audio networks to a natural image or speech signal. The resulting signals reflect the invariances instantiated in the network up to the matched layer. We then measured whether model metamers were recognizable to human observers – a necessary condition for the model representations to replicate those of humans.

Deep neural networks have been embraced as models of sensory systems, instantiating representational transformations that appear to resemble those in the visual and auditory systems. To more thoroughly investigate their similarity to biological systems, we synthesized model metamers – stimuli that produce the same responses at some stage of a network's representation. We generated model metamers for natural stimuli by performing gradient descent on a noise signal, matching the responses of individual layers of image and audio networks to a natural image or speech signal. The resulting signals reflect the invariances instantiated in the network up to the matched layer. We then measured whether model metamers were recognizable to human observers – a necessary condition for the model representations to replicate those of humans.